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ABSTRACT 



An independent checkpointing method using a memory 
checkpoint on a distributed system that includes a message 
transmission routine, a message processing routine, and a 
periodical checkpoint routine. The message transmission 
routine adds a self checkpoint number to a message to be 
transmitted when a current process tries to send a message 
to another process. The message processing routine per- 
forms a memory checkpoint and processes a message in 
reference to a checkpoint number of a transmission process, 
a checkpoint number of the current process, a memory 
checkpoint flag, and a message transmission flag when a 
message is received from a process. The periodical check- 
point routine performs a checkpoint that records a necessary 
state information for recovery against faults periodically in 
reference to the memory checkpoint flag. 

4 Claims, 4 Drawing Sheets 
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INDEPENDENT CHECKPOINTING METHOD 
USING A MEMORY CHECiCPOINT ON A 
DISTRIBUTED SYSTEM 



TECHNICAL FIELD 

The present invention relates to an independent check- 
pointing method, which recovers problems in running a 
distributed system and reduces work time. In the method in 
accordance with an embodiment of the present invention, 
transmitting processes send messages with a self checkpoint 
number, and receiving processes determine whether 
memory checkpoint is to be performed in reference to the 
transmitting processes' checkpoint number, current process 
checkpoint number, memory checkpoint flag, and message 
transmission flag before processing the received message. In 
performing periodical checkpointings, the method makes 
reference to the result of the memory checkpoint for check- 
pointing, 

BACKGROUND OF THE INVENTION 

Checkpointing technology stores status information of 
each process and recovers errors by using the stored infor- 
mation when errors occur in running distributed system 
environments. 

Several studies have been done in the area of checkpoint- 
ing technology. An adaptive checkpointing algorithm was 
proposed by Jian Xu et al. at Proceedings of Fifth IEEE 
Symposium on Parallel and Distributed Processing in 1993. 
The adaptive checkpointing algorithm performs checkpoint- 
ing by checking whether zigzag cycles made by an input 
message exist. If input messages make zigzag cycles, the 
adaptive checkpointing algorithm is based upon the fact that 
a domino effect may be caused. Therefore, if input messages 
make zigzag cycles, the adaptive checkpointing algorithm 
performs checkpointing to remove zigzag cycles before the 
processing of messages. 

A lazy checkpointing algorithm was proposed by Wang et 
al. at Technical report CRHC-92-27. In the lazy checkpoint- 
ing algorithm, message transmission processes attach a self 
checkpointing number to messages to be transmitted, and 
the message receiving processes compare the checkpointing 
number of the transmitted messages with their checkpoint 
number before processing the transmitted messages. If the 
checkpoint number of the transmission process is larger, 
checkpointing is performed before the message is processed. 
At the moment, the checkpoint numbers of the two processes 
become identical. 

However, when the amount of message transmission and 
execution speed difference increases, these checkpointing 
methods may cause a large number of checkpoints and 
increase job completion time. 

The number of checkpoints is directly related with job 
completion time in the error-free environment and the roll 
back distance is directly related with job completion time in 
the environment with errors. Therefore, a large number of 
checkpoints and an increase in roll back distance causes 
delayed job completion time. 

SUMMARY OF THE INVENTION 

An independent checkpointing method using memory 
checkpoint on a distributed system is provided. 

The independent checkpointing method in accordance 
with an embodiment of the present invention includes a 
message transmission routine, a message processing routine. 
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and a periodical checkpoint routine. The message transmis- 
sion routine adds a self checkpoint number to a message to 
be transmitted when a current process tries to send a 
message to another process. The message processing routine 

5 performs a memory checkpoint and processes a message in 
reference to a checkpoint number of a transmission process, 
a checkpoint number of the current process, a memory 
checkpoint flag, and a message transmission flag when a 
message is received from a process. The periodical check- 

10 point routine performs a checkpoint that records a necessary 
state information for recovery against faults periodica Uy in 
reference to the memory checkpoint flag. 

Preferably, the message transmission routine includes the 
following steps. A step is to generate a message to be 

15 transmitted. Another step is to add the checkpoint number of 
the current process to the message to be transmitted. A 
further step is to set the message transmission flag true for 
preparing cases in which an orphan message occurs. An 
additional step is to transmit the message. 

20 Preferably, the message processing routine includes the 
following steps. A first step is to receive the message from 
the process and compare the checkpoint number of the 
transmission process with the checkpoint number of the 
current process. Another step is to process the received 

25 message if the checkpoint number of the transmission pro- 
cess is smaller than or equal to the checkpoint number of the 
current process. A further step is to check the memory 
checkpoint flag if the checkpoint number of the transmission 
process is larger than the checkpoint number of the current 

30 process. Another step is to replace the checkpoint number of 
the current process with the checkpoint number of the 
message transmission process and process the received 
message if the checked memory checkpoint flag is true. A 
further step is to check the message transmission flag if the 

35 checked memory checkpoint flag is false. An additional step 
is to record the state information of the current process into 
a memory, set the checkpoint number of the current process 
as the checkpoint number of message transmission process, 
set the memory checkpoint flag true, set the message irans- 

40 mission flag false, and process the received message if the 
checked message transmission flag is true. Another addi- 
tional step is to replace the checkpoint number of the current 
process with the checkpoint number of the transmission 
process and process the received if the checked message 

45 transmission flag is false. 

Preferably, the periodical checkpoint routine includes the 
following steps. An initial step is lo check the memory 
checkpoint flag on a periodical checkpoint time. Another 
step is to record the state information stored at the memory 

50 to a disk if the checked memory checkpoint flag is true or 
recording the slate information to a disk and increasing the 
checkpoint number by one if the checked memory check- 
point flag is false. A further step is to calculate a next 
periodical checkpoint time. An additional step is to set the 

55 memory checkpoint flag and the message transmission flag 
false. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The embodiments of the present invention will be 
60 explained with reference to the accompanying drawings, in 
which: 

FIG. 1 is a diagram illustrating a process in a distributed 
system in accordance with an embodiment of the present 
invention; 

65 FIG. 2 is a flow diagram illustrating a message transmis- 
sion routine in accordance with an embodiment of the 
present invention; 
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FIG. 3 is a flow diagram illustrating a message processing current message transmission flag of the message receiving 

routine in accordance with an embodiment of the present process is true. 

invention; and FIG. 3 is a flow diagram illustrating a message processing 

FIG. 4 is a flow diagram illustrating a periodical check- ^0^^^"^ in accordance with an embodiment of the present 

pointing routine in accordance with an embodiment of the ^ invention. 

present invention. ^ shown in FIG. 3, the message processing routine 

includes the following seven steps. A first step is to receive 

DETAILED DESCRIPTION OF THE ^ message from a process and compare a checkpoint number 

INVENTION of the message transmission process with a checkpoint 

IQ number of the current process. A second step is to process 

The method in accordance with an embodiment of the the received message if the checkpoint number of the 

present invention may be applied to a distributed system, message transmission process is smaller than or equal to the 

which includes a number of local machines connected checkpoint number of the current process. A third step is to 

through LANs (local area network). check a memory checkpoint flag if the checkpoint number of 

FIG. 1 is a diagram illustrating a process in a distributed 15 message transmission process is larger than the check- 
system in accordance with an embodiment of the present ''''^^[ '^^^"t P^5^f • ^ fo"^^^ step is to 
invention. As shown in HG. 1. a local machine includes 'f^^^^ checkpomt number of the current process with 
local memory and a local disk. A local machine executes a the checkpoint number of the message transmission process 
process allocated to a job. Each process includes message process the received message if the memory checkpoint 
transmission routine, message processing routine, and peri- 20 <^hecked is tnie. A fifth step is to check the message 
odical checkpoint routine for recovery against faults. Each emission flag if the memory checkpoint flag checked is 
process sends information to other processes through mes- false. Asixth step is to record state mformation of the current 

process into memory, set the checkpomt number of the 

„ ' ^ . „ .„ . . current process as a checkpoint number of message trans- 

FIG. 2 IS a flow dia^atn lUustratuig a message transm^- ^^^^^ ^ checkpoint flag true, set the 

sion routine in accordance with an embodiment of the 25 transmission flag false, and process the received 

present invention As shown in FIG. 2, the independent ^ ^ (he message transmission flag checked is true. A 

checkpomting method includes four steps. A firet step is to ^^yg^.i, ^ ^^ ^^^^ checkpoint number of the 

generate a message to be transmitted. A second step is to ^„„^„t process with a checkpoint number of the message 

attach checkpomt number of current process to the message. transmission process and process the received message if the 

A third step IS to set transmission flag true for processing 30 ^ transmission flag checked is false, 

orphan messages. A fourth step is to transmit the message. • j r .t. 

" When a message is received trom other processes, the 

When the current process is running and a new message message processing routine compares the checkpoint num- 

needs to be transmitted to a different process runnmg in a ^he current process with the checkpoint number of the 

different local machme, a message that needs to be trans- message transmission process that is added to the message 

mitted earlier is generated at step SIO. ^5 ^^^^^^ processing the received message at step S21 and S22. 

Then, a checkpoint number of the current process is added if the checkpoint number of the message transmission 

to the generated message to be transmitted at step Sll. process is smaller than or equal to the checkpoint number of 

Executing a periodical checkpoint routine that is to be the current process, it means that the processing speed of the 

described later may increase the checkpoint number of the message transmission process is slower than or equal to the 

current process. When a message arrives from other processing speed of the current process. In such cases, since 

processes, the checkpoint number of the current process may the memory checkpoint of the current process is not 

be replaced by the checkpoint number of the transmission necessary, the current process treats the received message 

process. without performing the memory checkpoint at step S25. 

After the checkpoint number is added to the message to be 45 However, if the checkpoint number of the message trans- 
transmitted, the message transmission flag is set true for mission process is larger than the checkpoint number of the 
notifying that the current process has sent a message to current process, it means that the processing speed of the 
another process and then the message is transmitted at step message transmission process is faster than the processing 
S12 and S13. speed of the current process. That is, the current process 

The message transmission flag is important because it is 50 determines whether the memory checkpoint is to be per- 

neccssary to reduce the number of the memory checkpoint formed before processing the received message, 

when orphan messages are generated. Orphan messages are The memory checkpoint is performed only if the check- 

ihe messages transmitted after the checkpoint of the message point number of the message transmission process is larger 

transmission processes and received before the checkpoint than the checkpoint number of the current process, the 

of the message receiving processes. When errors occur in the 55 memory checkpoint has not been performed, and the 

message transmission processes, orphan messages bring memory checkpoint after transmitting a message to other 

back the message receiving processes to the point before the processes and periodical checkpoint have not been per- 

checkpoints. Therefore, in order to avoid generating orphan formed. It reduces number of checkpoints by preventing 

messages, checkpointing is performed after the message repetition of memory checkpoints and therefore reduces job 

transmission processes send a message. go completion time, even if a number of messages arc received 

The method in accordance with an embodiment of the from other processes before the periodical checkpoint is 

present invention employs message transmission flags to performed. 

reduce the number of memory checkpoints in overall execu- For this purpose, the memory checkpoint flag is checked 

tion of each process and to prevent orphan messages. That to see whether the memory checkpoint has been performed 

is, memory checkpoint is performed only if the checkpoint 65 before at step S23. 

number of the message transmission process is larger than If the flag is true, which means the memory checkpoint 

checkpoint number of the message receiving process and the was performed after the periodical checkpoint is performed, 
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the memory checkpoint is not to be performed. Instead, the That is, process state information is recorded to disks with 

checkpoint number of the current process is replaced by a a certain period. Even though memory checkpoint is caused 

checkpoint number of the message transmission process and in every period, the time for memory checkpoint is substan- 

then a next message is processed at steps S24 and S25. tially smaller than the time for the disk checkpoint, which 

If the flag is false at step S23, the message transmission 5 means that actual job delay time is trivial, 

flag is confirmed at step S26. On the contrary, to the conventional checkpoint methods, 

The message transmission flag is a flag describing if the the method in accordance with an embodiment of the present 

current process has sent any messages to other processes. invention performs only one disk checkpoint with a certain 

The message transmission flag is set as true by the message period. 

transmission routine and as false after the memory check- lo addition, as shown in FIG. 3, the method in accordance 

point or periodical checkpoint is performed. with an embodiment of the present invention removes 

Therefore, if the message transmission flag is true, which dependent relations between processes that are usually 
means that the result of the ^ep S26 is true, that is, memory caused by message transmissions by comparing checkpoint 
checkpoint or periodical checkpoint is not performed yet, numbers. That is, when a process receives a message with a 
the memory checkpoint that stores the state information of larger checkpoint number than the process, the process 
the current process into the memory is performed. Check- performs the memory checkpoint before processing the 
point numbers become identical by replacing checkpoint message and identifies a checkpoint number with the trans- 
number by checkpoint number of message transmission mission process. This procedure virtually removes diSer- 
proccss at step S27. ence of execution speeds and reduces dependent relations 

Then, the memory checkpoint flag is set true and the between processes. As a result, the method in accordance 

message transmission flag value is set false at steps S2S and with an embodiment of the present invention has a short roll 

S25 before the next message is processed. back distance when errors occur. 

HG. 4 is a flow diagram illustrating a periodical check- As described above, the method of the present invention 
pointing routine in accordance with an embodiment of the adds a checkpoint number in transmitting messages between 
present invention. As shown in FIG. 4, the periodical check- processes. When a checkpoint number of message transmis- 
pointing routine includes the following steps. A first step is sion process is larger than a checkpoint number of a current 
to check the memory checkpoint flag on periodical check- process, the method of the present invention performs the 
point time. A second step is to record process state infor- memory checkpoint in reference to a memory checkpoint 
mation stored at memory to disk if the memory checkpoint flag and message transmission flag, replaces the checkpoint 
flag checked is true or recording process state information to number of the current process with a checkpoint number of 
disk and increasing the checkpoint number by one if the the message transmission process. In addition, the method of 
memory checkpoint flag checked is false. A third step is to the present invention uses the result of the memory check- 
calculate a next periodical checkpoint time. A fourth step is point in performing a periodical checkpoint, 
to set the memory checkpoint flag and the message trans- Therefore, when a number of processes perform the same 
mission flag false. jobs in a distributed system environment, the method in 

The periodical checkpoint routine checks whether the accordance of an embodiment of the present invention has 

memory checkpoint flag is true or false before writing minimum number of checkpoints regardless of the amount 

current state information of the process with a certain period of message transmission and execution speed difference, 

at steps S31 and S32. ^ Also, the method removes dependent relations between 

The memory checkpoint flag is a flag indicating the processes that are usuaUy caused by message transmissions 

memory checkpoint has been performed in the message by comparing checkpoint numbers. The method in accor- 

processing routine. If the memory checkpoint flag is true, dance with an embodiment of the present invention has a 

which means the memory checkpoint has been performed short roll back distance when errors occur. Consequently, the 

recently, the process state information stored in memory is method is effective for completing jobs in a minimum 

written to the disk without increasing the checkpoint number amount of time in a distributed system environment with 

at step S32. errors or in a distributed system environment without errors. 

If the memory checkpoint flag is false, which means the Although representative embodiments of the present 

memory checkpoint has not been performed recently, the invention have been disclosed for iflustrative purpose, those 

process state information is written on the disk while jg who are skilled in the art will appreciate that various 

increasing the checkpoint number by 1 at step S33. modifications, additions and substitutions are possible with- 

Then, the amount of time for executing the periodical out departing from the scope and spirit of the present 

checkpoint is calculated and the memory checkpoint flag invention as defined in the accompanying claims and in 

and message transmission flag are set as false at steps S34 equivalents thereof, 

through S36. 55 What we claim: 

When the amount of message transmission and execution 1. An independent checkpointing method using a memory 
speed difference increase, conventional checkpointing meth- checkpoint on a distributed system, comprising: 
ods cause a large number of checkpoints, which increases a message transmission routine for adding a self check- 
delay caused by checkpoints and eventually causes point number to a message to be transmitted when a 
increased job completion time. 60 current process tries to send a message to another 

However, the method in accordance with an embodiment process; 
of the present invention only performs checkpoint on the a message processing routine for performing a memory 
memory when a message is initially received from a process checkpoint and processing a message in reference to a 
with a larger checkpoint number. Later, the method increases checkpoint number of a transmission process, a check- 
only the checkpoint number without performing any check- 65 point number of the current process, a memory check- 
points. As shown in FIG. 4, process state information stored point flag, and a message transmission flag when a 
in the memory is recorded to disks in periodical checkpoints. message is received from a process; and 
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a periodical checkpoint routine for performing a check- 
point that records a necessary state information for 
recovery against faults periodically in reference to the 
memory checkpoint flag. 

2. The independent checkpointing method of claim 1, 5 
wherein the message transmission routine comprises the 
steps of: 

generating the message to be transmitted; 
adding the checkpoint number of the current process to 
the message; 

setting the message transmission flag to true for preparing 

cases in which an orphan message occurs; and 
transmitting the message. 

3. The independent checkpointing method of claim 1, 15 
wherein the message processing routine comprises the steps 
of: 

receiving the message from the process and comparing 
the checkpoint number of the transmission process with 
the checkpoint number of the current process; 20 

processing the received message if the checkpoint number 
of the transmission process is smaller than or equal to 
the checkpoint number of the current process; 

checking the memory checkpoint flag if the checkpoint 
number of the transmission process is larger than the 
checkpoint number of the current process; 

replacing the checkpoint number of the current process 
with the checkpoint number of the message transmis- 
sion process and processing the received message if the 
checked memory checkpoint flag is true; 
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checking the message transmission flag if the checked 
memory checkpoint flag is false; 

recording the state information of the current process into 
a memory, setting the checkpoint number of the current 
process as the checkpoint number of message trans- 
mission process, setting the memory checkpoint flag 
true, setting the message transmission flag false, and 
processing the received message if the checked mes- 
sage transmission flag is true; and 

replacing the checkpoint number of the current process 
with the checkpoint number of the message transmis- 
sion process and processing the received message if the 
checked message transmission flag is false. 

4, The independent checkpointing method of claim 3, 
wherein the periodical checkpoint routine comprises the 
steps of: 

checking the memory checkpoint flag on a periodical 
checkpoint time; 

recording the state information stored in the memory to a 
disk if the checked memory checkpoint flag is true or 
recording the state information to a disk and increasing 
the checkpoint number by one if the checked memory 
checkpoint flag is false; 

calculating a next periodical checkpoint time; and 

setting the memory checkpoint flag and the message 
transmission flag false. 

« « « « « 
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